Skip to content

fix(cli): stats told operators to set a variable that does not exist - #160

Merged
blitzcrieg1 merged 1 commit into
masterfrom
fix/cli-stats-env-var
Sep 3, 2026
Merged

fix(cli): stats told operators to set a variable that does not exist#160
blitzcrieg1 merged 1 commit into
masterfrom
fix/cli-stats-env-var

Conversation

@blitzcrieg1

Copy link
Copy Markdown
Owner

The bug

agentmetry stats answered a disabled audit export with:

Audit export disabled, enable AGENTMETRY_AUDIT_EXPORT to collect stats.

There is no such setting. The only alias is AGENTMETRY_AUDIT_EXPORT_ENABLED,
declared on Settings in core/config.py. AGENTMETRY_AUDIT_EXPORT appeared
exactly once in the entire codebase: inside the string telling somebody to set
it.

The cost is the worst shape this project has. The operator reads the message,
sets the variable, restarts, and nothing changes. The trail stays empty and
nothing says why. A recorder that is off while its operator believes it is on is
the failure the product exists to prevent.

How it was found

By contrast, not by testing. #139
added the same message to agentmetry detections and named the variable
correctly. Reviewing that commit is the only reason anyone read the older one.

The guard

A string comparison would pin this instance and miss the next. So
test_cli_env_var_names.py walks every string literal the CLI prints, extracts
anything shaped like an AGENTMETRY_ variable, and checks it against what the
package actually reads.

"Actually reads" needed both halves. The aliases declared on Settings are not
the whole set: AGENTMETRY_AGT_HMAC_KEY is read straight from os.environ at
the point of use and never passes through Settings. The first version of this
test flagged it as fake. Getting that right is what stops the guard from being
wrong in the opposite direction and training people to ignore it.

Verified both ways: the test passes on this branch, and reintroducing the old
name fails both assertions.

Checks

Nothing frozen is touched, fingerprint unchanged at 15846a0915769d4a. 1,192
tests pass, ruff clean, benchmark 0 missed and 0 false positives.

`agentmetry stats` answered a disabled audit export with "enable
AGENTMETRY_AUDIT_EXPORT". There is no such setting. The only alias is
AGENTMETRY_AUDIT_EXPORT_ENABLED, so the name appeared exactly once in the whole
codebase: in the string telling somebody to set it.

The cost is the worst one this project has: the operator follows the
instruction, the variable does nothing, the trail stays empty, and nothing says
why. A recorder that is off while the operator believes it is on is the failure
the product exists to prevent.

Found by contrast rather than by testing. #139 added the same message to
`agentmetry detections`, named the variable correctly, and only then did anyone
read the older one.

So the guard is not a string comparison. `test_cli_env_var_names.py` walks every
string literal the CLI prints, pulls out anything shaped like an AGENTMETRY_
variable, and checks it against what the package actually reads: the aliases
declared on `Settings`, plus names read straight from `os.environ`, because
AGENTMETRY_AGT_HMAC_KEY is real and never passes through Settings. Writing that
second half is what stopped this test from being wrong in the other direction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@blitzcrieg1
blitzcrieg1 merged commit 610c370 into master Sep 3, 2026
9 checks passed
@blitzcrieg1
blitzcrieg1 deleted the fix/cli-stats-env-var branch September 3, 2026 16:16
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 3, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant